An investigation into the possibility of using sociological research methodologies for the elicitation of tacit knowledge for building knowledge intensive systems

ion than domain discourse, e.g. an accountant and an engineer can converse in terms of the basic assumptions of science (objective facts, rules, etc.) and mathematical reasoning. Both engineering and accountancy are extremely different domains but both share common assumptions. While this is extremely useful for inter-disciplinary discourse it is problematic for knowledge elicitation of tacit knowledge. If interviews between experts and engineers take place at a higher level of abstraction the knowledge elicited will tend to be more formal domain level. In fact, if a discourse is entered into with scientific assumptions and mathematical reasoning it is difficult to imagine how topics external to the formal domain knowledge can be discussed. The medicine discipline (a topic pertinent to this research) is an example which will illustrate this point. Medicine is considered a scientific subject (comparable to applied biology), however, the practice of medicine has a very important nonscientific aspect which is often tacitly held. Byrne and Long (1976) recorded 2500 consultations with G.P.s and concluded that approximately 75% of the interviews were 'doctor centred'. The doctor would ask a series of closed questions about the patients symptoms (e.g. 'When did you last move your bowels ?' 'Did you experience any pain ?' etc.) and in some cases physically examine the patient prior to diagnosis. If a doctor was asked by a knowledge engineer, (a non-doctor) how s/he diagnosed a particular illness s/he would be likely to reiterate the above consultation. This could be encoded, e.g. IF the patient has not moved their bowels for 3 days and when they did it was painful THEN illness Z. A. W. Henshall, Information Technology Institute, University of Salford Page 103 P/iD Thesis C'hapter 8. Research Methods The code could later be checked by the doctor for verification. The code would be found to be correct because it corresponded to medical knowledge. However, in practice the process of diagnosis is much more complex. 8.3.2.1. 1.2 Medical Knowledge & Clinical Experience A doctors diagnosis is the result of a combination of medical knowledge (formal knowledge) and his/her clinical experience (experiential knowledge). The importance of clinical experience in the training of doctors has been well documented. Foucault (1973) dates the beginning of modern medicine to the creation of the 'clinic' at the beginning of the 18th Century. The clinic allowed the systematic, large scale, studying and teaching of disease. The clinic witnessed the emergence of the clinician at the bedside of the patient. Dingwall and Heath (1977) note, 'The clinic emerged when it became possible to treat the individual as a field of investigation, and that space by the patient's bedside, therefore, became the locus of medical inqui,y and research, as well as treatment and instruction.' (page 86) From this point, a key component of the transmission of medical knowledge occurs at the bedside. Jamous and Peloille (1970) note although medicine has undergone considerable structural changes, the 'real work' of a doctor is taught by the bedside by an apprenticeship system. A doctor becomes a competent practitioner not in a library or laboratory (though each of these have their place) but at the bedside where they receive first hand, what Becker, Geer and Strauss (1961) refer to as 'clinical experience' treating patients. Becker, Geer and Strauss (1961) point out that there often occurs a conflict between 'clinical experience' and theoretical or scientific knowledge. Where such a conflict A. W. Henshall, Information Technolog y Institute, University of Salford Page 104 PhD Thesis Chapter 8. Research Methods occurs 'clinical experience' almost always predominates. Dingwall and Heath (1977) note, 'These authors go on to comment that "argument from experience was quite commonly used and considered unanswerable'. Such unanswerable experience is gained in the context of clinical instruction which is itself unquestionable.' (page 88). Therefore, although doctors appear to make diagnosis using objective formal scientific knowledge, they are also diagnosing by tacit use of their 'clinical experience. It is important to note the high status of clinical experience vis a vis medical knowledge. 8.3.2.1.2 Problem for Knowledge Elicitation It is relatively easy for a knowledge engineer to communicate with the expert at the penetrable horizons of their respective domains. This is a natural tendency in interdomain communication, however, if, for example this happens between a doctor and an knowledge engineer, they will have long discussions of cause and effect (often at a complex level) about the medical knowledge of the doctor. This can be advantage (it eliminates much of the confusing detail) if the aim is to elicit objective domain knowledge. However, the aim of the interview is to gain an understanding of the process of diagnosis, large areas of important knowledge such as 'clinical experience' will remain 'invisible'. The explication of tacit knowledge requires methods capable of penetrating deep into the centres of density of the experts field of interest. A. W. Henshall, Information Technology Institute, University of Salford Page 105 PhD Thesis Chapter 8. Research Methods 8.3.3 Problems of Asking Questions Many knowledge elicitation techniques revolve around asking questions, of various types (this includes questionnaires etc). However, the nature of tacit knowledge makes its explication through questioning problematic. As Polanyi (1966) notes that although we 'attend from' proximal terms, if we concentrate on (attend to) these terms they lose their meaning. Tacit knowledge is not normally consciously referred to by individuals, it remains unexamined and unexplained. If an engineer directly questions a respondent about a specific area of tacit knowledge, in order to reply the respondent must first consciously examine that knowledge. Once explicitly examined the tacit knowledge changes in character. 8.3.3.1 Tacit knowledge made Explicit Becomes Qualified To illustrate how tacit knowledge changes in character when consciously examined, the following example will be used. An individual could at an unexamined and unexplained level believe that 'all people with red hair are short tempered' and order his life accordingly. However, if asked the question 'Do all people with red hair have a short temper?' They are forced to examine this belief, once this occurs they can think of many cases where this is not true e.g. a) their next door neighbor, b) people who dye their hair red, c) their aunt who was always kind to them etc. Instead of replying, 'Yes all people with red hair are short tempered.' They are more likely to reply, 'Yes, people with red hair often have a short temper, except my next door neighbor, or if they dye their hair or are my kindly aunt. A. W. Henshall, Infor,nation Technology Institute, University of Salford Page 106 PhD Thesis Chapter 8. Research Methods The tacitly held assumptions are thus changed simpiy by making them known to the respondent. This experience changes the character of the proximal term, once examined it can no longer remain viable without the being qualified with a series of exemptions. This transformation makes the elicitation of tacit knowledge extremely complex, because each respondent is likely due to their different personal histories to provide different exemptions. Direct questioning transforms the character of tacit knowledge, therefore, knowledge acquisition techniques which directly question experts, are inappropriate for the elicitation of tacit knowledge. 8.4 Know kdge Elicitation Techiikiues that Do Not Ask Direct The above indicate that methods which seek objective domain knowledge, methods which depend on interviewing experts and methods which directly question experts are likely to be inappropriate for the elicitation of tacit knowledge. However, not all knowledge acquisition methods fall into these three categories, these methods will now be briefly discussed and assessed for the purpose of the elicitation of tacit knowledge. 8.4.1 Critical Incident Technique A knowledge elicitation technique which does not rely on direct questioning is Critical Incident Technique (CIT). Doukidis & Whitley (1988b) recommend this method for heuristic modelling and seems particularly relevant for this research because they contend, 'This technique enables: Definition, location and categorization of all basic concepts in the domain.' (page 78). A. W. Henshall, Information Technology Institute. University of Salford Page 107 PhD Thesis Chapter 8. Research Methods CIT is an applied psychology technique, first attributed to Flanagan (1954). It is used to elicit knowledge by asking experts to describe past incidents that they consider of particular interest (critical incidents). CIT has been applied to knowledge acquisition by Bliss & Ogborn (1979) and has been recommended for heuristic modelling Poulymemakou & Doukidis (1987). CIT is an interesting technique in that it allows the expert to, in some ways set the agenda by choosing interesting past events. The expert can chose incidents that the knowledge engineer might not recognize as 'interesting' thus raising questions, of which the knowledge engineer (left to his/her own resources) might not have thought. However, CIT is in many ways a formal interview in which once the expert has identified a critical incident s/he is invited to explain their actions step by step fashion. CIT may be useful in eliciting Heuristics and pinpointing critical areas of decision making, however, it takes the form of a 'post hoc rationalization' of an interesting incident . One in which much of the data has already been unconsciously 'written out' of the account by the respondent. By definition that which is left out, is that which does not need to be included, because it is self-evident. Unfortunately for this research what is self-evident to the members of a mini-culture is likely to contain the tacit knowledge. To use the example cited by Collins (1987) of the physicist Harrison attempted to build a TEA laser. In order to do this he received a set of instructions from a colleague (whom shall be referred to as Dr. X) who had previously built a working TEA Laser. If Dr. X had been asked to relate a critical incident in his career he might have cited the building of a TEA laser. When asked to explain how he had achieved this he would have presented a step by step set of instructions similar to those provided to Harrison. For problems associated with post hoc rationalization, see Coulter (1983) and Suchman (1985) above. A. W. Henshall, Information Technology Institute, University of Salford Page 108 PhD Thesis Chapter 8. Research Methods To recap, although Harrison succeeded in building a TEA laser using these instructions he could not get it to work. One instruction was that the wires to the capacitors should be 'short'. Harrison made them as short as possible, but, it finally emerged that for the wires to be 'short' in laser building terms, required the capacitors must be inverted in a frame. This point had been written out of the instruction because it was redundant, an instruction that Dr. X would tacitly assume any body constructing a TEA laser would know. If CIT was employed on the expert laser builder, his post hoc rationalization would have omitted the inversion of the capacitors, in the same way it would have omitted the information that nuts must be turned in a clockwise direction to tighten them on the bolt. This is problematic for the purposes of this research in that the knowledge omitted is likely to contain tacit knowledge. 8.4.1.1 Potential for the Elicitation of Tacit Knowledge Although CIT has many uses, it is too linear for the purposes of this research. Post hoc rationalization omits much of the tacit knowledge from the narrative, therefore, this method is of limited use to this research. 8.4.2 Protocol Analysis Protocol analysis attempts to overcome many of the problems of post hoc rationalizations of CIT, by eliciting knowledge, while the expert actually carries out his/her task. There are many variations on the technique, these will be classified under three main headings and assessed in relation to the elicitation of tacit knowledge. A. W. Henshall, Infor,nation Technology Institute, University of Salford Page 109 PhD Thesis Chapter 8. Research Methods 8.4.3 Variations of Protocol Analysis Protocol analysis is the method by which knowledge elicitation attempts to capture the complex data which is the basis of decision making. The general method of protocol analysis takes the form of an expert being given various problems to solve. This is usually video taped (although audio tape is sometimes used) and the tapes later analyzed for the protocols the expert employs, when making expert decisions. There are many variations of protocol analysis, these fall into three main categories, which will now be assessed for their applicability for the elicitation of tacit knowledge. 8.4.3.1 Concurrent Protocols To conduct concurrent protocol analysis, the expert is set a series of domain problems and asked to 'think out loud' as s/he solves the problems. The thinking out loud is then later analyzed for decision protocols. This appears to allow the expert to make decisions in as realistic a way as possible. There are, however, several drawbacks to this method: the Hawthorne Effect must be noted; transcripts with large irrelevant areas need to be analyzed; the choice of problem will affect the resulting protocol. 8.4.3.2 Restricted Protocols To overcome the irrelevant material the expert can be asked to restrict his responses to specific areas. This results in shorter, more easily analyzed protocols. However, this method tends to force the respondent to concentrate on his responses, rather than 'thinking out loud'. This could affect the objectivity of the responses. A. W. Henshall. Information Technology Institute, University of Salford Page 110 PhD Thesis Chapter 8. Research Methods 8.4.3.3 Retrospective Protocols In the final variation of protocol analysis the expert is set a series of problems, after completing the tasks, he is asked to explain the protocols by which he completed the task. This method presents the problem associated with post hoc rationalizations. The respondent will often construct a reasonable account of why they made certain actions, the relationship of this account and the actual actions is at best dubious (see Coulter (1985) in Chapter Two). 8.4.3.4 Matching the Type of Protocol Analysis to the Task The different methods of protocol analysis have strengths and weaknesses in respect to different research objectives: Concurrent Protocols: Best suited to research into the decision making process and to explore ill defined research areas. Restricted Protocols: Best suited to research where problem areas have already been identified. Retrospective Protocols: Best suited to testing hypothesis regarding information content. Ericsson and Simon (1980) after studying a large number of cases to assess the effect of verbalization on the task being performed found. "verbalizing information is shown to affect cognitive processes only if the instructions require verbalization of information that would not otherwise be attended to" (p.215). A. W. Henshall, Information Technology Institute, University of Sniford Page 111 PhD Thesis Chapter 8. Research Methods Therefore, concurrent protocols are much more likely to capture more accurately the decision process. Interference by the researcher in restricting or focussing the respondent, while simplifying the analysis, is likely to adversely affect the results of the analysis. Concurrent protocol analysis can add depth to purely objective types of knowledge elicitation. For example, Bainbridge et al (1969) set a group of experts and a group of novices a task. To the surprise of the researchers, when measured quantitatively both groups produced similar results. However, when re-tested it was found that, 'The experienced group was setting themselves a harder task by tlying to take into account more factors, the result being that their peiformance was no better on the simulation.' (page 98) 8.4.3.5 Potential for the Elicitation of Tacit Knowledge Concurrent protocol analysis appears with careful analysis to be capable of capturing tacit knowledge. The expert by 'thinking out loud' could reveal tacit knowledge essential to the decision making process. However, this method does not seem practical for this research for several reasons. 8.4.4 Practical Considerations Protocol analysis is usually used to trace the decision making process of experts. This assumes that decisions are made with in limited, framework. For example Hayes (1988) used protocol analysis in order to elicit the way international bank credit officers decide which businesses to grant loans. Concurrent protocol analysis was the chosen research method. This was conducted in the field to make the analysis as realistic as possible. However, rather than asking the participants to think out loud as they made real decisions, the concurrent protocol analysis was conducted A. W. Heashall, Information Technology Institute, University of Salford Page 112 PhD Thesis Chapter 8. Research Methods in an adjacent room (so that the loan officers would not be disturbed by phone calls), using an artificial test case. The test case was based on figures from a liquor firm, Distillers, which had just purchased a distribution company in New York. The test case changed the name of the firm to Waperton-Smith (in order that the officers did not have personal knowledge of the firm), the turnover was divided by 100 (in order that the loan would be of the size the officers were used to dealing with), the business was changed from liquor to liqueurs and the distribution company was situated in California. None of these changes were felt to change the decision making process. Although this field test was set up to be as realistic as possible, any one of these changes could have a dramatic effect on the decisions the officers reached. Moving the officer into a room without a phone, on the one hand eliminates interruptions, however, it also isolates the officer from his contacts with the outside world. Such contacts (for example if the officer had a colleague with an expert knowledge of the liqueur business) could drastically effect the officers decision. Changing the name of the firm indicates Hayes believes that all firm are interchangeable. Firms have diverse objectives, the purchase of a distribution company might make economic sense to one firm and not to another. The officer might know or have access to the objectives of particular Firms. The liquor and the liqueur business can be considered interchangeable only if the officer knows nothing of either business. Likewise changing the site of the distribution company from New York to California might have a dramatic effect upon whether it was a good buy. Finally to assume what is economic sense to company X, would also make economic sense to company Y, 100 times its size, seems to be, at least naive. If a desk based, purely decision making process can not be captured by protocol analysis without such drastic modifications to the experts usual behavior, the technique is likely to be totally impractical in a multi-function (midwives and doctors A. W. Henshall, Information Technology Institute, University of Salford Page 11 3 PhD Thesis Chapter 8. Research Methods perform many more tasks than decision making), mobile (doctors and midwives are not confined to desks) dynamic (the work of the maternity unit changes minute by minute) process such as a maternity unit. Protocol analysis might be useful in limited well defined areas, but for the main field work other research methods must be sought. The above seems to indicate the research techniques of science and technology in general, and knowledge acquisition in particular are inappropriate for the elicitation of tacit knowledge. Therefore, the research methods of other overlapping disciplines will now be investigated to assess their usefulness for this research. Investigations of this type are extremely worthwhile, because the identification of research methods from one domain which can be applied in another domain, is extremely valuable. 8.5 Sociok gical Research \ I ethods One such overlapping domain is identified by Collins (1987) who contends that sociologists and knowledge engineers share a topic: the explication of cultural knowledge. Sociologists can help knowledge engineers in this task by contributing the findings of both philosophy and the 'modern sociology of knowledge'. 8.5.1 Sociology and Knowledge Acquisition Share a Topic Collins (1987) contends that sociologists and knowledge engineers share a topic: the explication of cultural knowledge. For Collins progress in what he refers to as Intelligent Knowledge Based Systems (IKBS), will depend on progress in explicating the cultural knowledge base and making the culture of the expert, understandable to the less skilled end user. A. W. Henshall, Information Technology Institute, University of Salford Page 114 PhD Thesis Chapter 8. Research Methods Collins (1987) arrives at this conclusion by assessing recent developments in IT. Collins notes that most of the debates in Al have centered around the dichotomy of the algorithmic and enculturational models of knowledge. 8.5.1.1 Algorithmic Model The algorithmic model holds knowledge is transferable via logical instructions, as a kind of 'recipe'. This notion is central to the practice of 'replication', a fundamental plank in the scientific method. 8.5.1.2 Enculturational Model The enculturational model holds that knowledge is only transferred by a method which is closer to 'learning a skill'. This model holds that knowledge is difficult to transfer by logical instruction because it contains a 'tacit' component. Collins (1987) discusses each model by reference to a previous study of replication (Collins 1974), and concludes that the enculturational model was correct. Collins contends that it is only by joining a culture, (or more correctly a mini-culture) that one can take on board the taken-for-granted-assumptions which members of the miniculture routinely use to interpret the world. Collins (1987) demonstrated for knowledge to be successfully transferred required, the transfer of cultural knowledge. Collins maintains that algorithms alone are not sufficient for the transfer knowledge because of the five propositions for the transference of expertise. A. W. Henshall, Information Technology Institute, University of Salford Page 1 13 PhD Thesis Chapter 8. Research Methods COLLINS' FIVE PROPOSITIONS FOR THE TRANSFER OF KNOWLEDGE Proposition 1: